Accelerating mini-batch SARAH by step size rules

نویسندگان

چکیده

StochAstic Recursive grAdient algoritHm (SARAH), originally proposed for convex optimization and also proven to be effective general nonconvex optimization, has received great attention because of its simple recursive framework updating stochastic gradient estimates. The performance SARAH significantly depends on the choice step size sequence. However, variants often manually select a best-tuned size, which is time consuming in practice. Motivated by this gap, we propose variant Barzilai-Borwein (BB) method, referred as Random (RBB) determine mini-batch setting, leading new method: MB-SARAH-RBB. We prove that MB-SARAH-RBB converges linearly expectation strongly objective functions. Moreover, analyze complexity show it better than original method. To further confirm efficacy RBB MB-SARAH+-RBB incorporating into MB-SARAH + Numerical experiments standard data sets indicate our methods outperform or match state-of-the-art algorithms.

برای دانلود باید عضویت طلایی داشته باشید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Distributed Mini-Batch SDCA

We present an improved analysis of mini-batched stochastic dual coordinate ascent for regularized empirical loss minimization (i.e. SVM and SVM-type objectives). Our analysis allows for flexible sampling schemes, including where data is distribute across machines, and combines a dependence on the smoothness of the loss and/or the data spread (measured through the spectral norm).

متن کامل

Multiple Batch Sizing through Batch Size Smoothing

Batch sizing in different planning period is categorized as a classical problem in production planning, that so many exact & heuristic methods have been proposed to solve this problem, each of which considering various aspects of the original problem. The solution obtained from majority – e.g. MRP – is in this format that there may be some periods of idleness or each period should produce a...

متن کامل

Nested Mini-Batch K-Means

A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested mini-batches, whereby data in a mini-batch at iteration t is automatically re...

متن کامل

Turbocharging Mini-Batch K-Means

A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested mini-batches, whereby data in a mini-batch at iteration t is automatically re...

متن کامل

multiple batch sizing through batch size smoothing

batch sizing in different planning period is categorized as a classical problem in production planning, that so many exact & heuristic methods have been proposed to solve this problem, each of which considering various aspects of the original problem. the solution obtained from majority – e.g. mrp – is in this format that there may be some periods of idleness or each period should produ...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: Information Sciences

سال: 2021

ISSN: ['0020-0255', '1872-6291']

DOI: https://doi.org/10.1016/j.ins.2020.12.075